• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    The invention describes an immersive interaction method using an interactive mirror surface that enables a visual contact dialog with one or a plurality of users. The process is composed of the repetition of steps (a, b and c). Step (a) performs the media acquisition (f) and processes the man-machine interactions. Step (b) builds the visual contact (s) using a custom mirror reflection. Step (c) renders the audio-visual rendering and contextualizes it.
    公开号:CH711803A2
    申请号:CH01702/15
    申请日:2015-11-23
    公开日:2017-05-31
    发明作者:Van Kommer Robert
    申请人:Kiodia Sàrl；
    IPC主号:
    专利说明:

    Description
    TECHNICAL AREA
    The invention relates to a method of immersive interactions and computer vision applied to a virtual mirror capable of creating a dialogue by visual contact with one or a plurality of users.
    STATE OF THE ART
    The invention of the mirror is attributed to the German chemist Justus von Liebig in the year 1835, the well-known manufacturing process is to apply a metal film on a transparent glass surface. The principle of reflection of light is governed by the laws of physics and, therefore, physical mirrors have a set of strong limitations that result from it, such as the fact that the reflected image is inverted with respect to reality which makes a text reflected difficult to read. Although the use of physical mirrors is still widespread, other so-called virtual mirror systems have come into being by combining a video camera, a computer processing system, and a display on a screen. More recently, new virtual mirror systems have added augmented reality elements to give people the opportunity to try, for example, a new hairstyle or new clothes. More recently, some virtual mirrors offer features and 3D representations.
    [0003] Virtual 3D mirror systems already exist in the state of the art.
    [0004] Patent US 2015 120 496 (A1) describes a clothing purchase system that makes it possible to reconstruct a portion of the human body in 3D and then to virtually try on a suit. The system uses a virtual mirror and several shots. In one embodiment, the user can select the point of view of one of the shots and even an intermediate angle between several shots.
    US Patent 2014 232 816 discloses a telepresence system composed of several systems interconnected by a communication network. The metaphor of the semi-transparent mirror is used to allow participants to see each other and to see the distant interlocutors as if they were in the same room, that is to say that the image projected on the mirror is a reconstruction 3D of the remote environment. The visual contact between participants can thus be preserved and additional information can also be visualized.
    [0006] US Patent No. 2,226,000, which describes a virtual mirror capable of detecting the presence of a user, of visualizing his image adequately by resizing this one with the main objective of recognizing the user among those registered in a database and this while imitating the behavior of a physical mirror.
    [0007] Most of these known 3D virtual mirror solutions are therefore based on an extension of the functionalities of a virtual mirror by adding 3D vision, additional information, and in certain cases additional augmented reality elements that faithfully follow. the movements of the person.
    A major flaw in the proposed physical and virtual mirrors and interactive systems in general is that they do not provide the ability to establish and react to visual contact with the interaction surface. This disadvantage has the consequence that these surfaces do not have the ability to actively attract the attention of a potential user or to interact with him through visual contact. In addition, when the system is used in spaces where multiple users can potentially interact, other users can not see who the particular system communicates with.
    As regards the physical mirrors, these are governed by the laws of physics and are therefore limited by these constraints. They only offer the functionality of a mirror, while occupying an imposing surface.
    The physical mirrors and virtual mirrors described in the state of technology are not intended to establish active visual contact between a user and the mirror itself, therefore, they can not validly attract the user. the visual attention of a particular user, nor does it involve an immersive interaction when the user is standing nearby.
    Other immersive interactions use helmets and other specific accessories and are therefore not suitable for communication spaces where the interaction is spontaneous and ephemeral. In addition, these systems do not allow to be seen or used by a plurality of users.
    It would therefore be advantageous for the mirror surfaces to be interactive and to be able to dialogue by visual contact with one or a plurality of users.
    BRIEF SUMMARY OF THE INVENTION
    An object of the present invention is to overcome the disadvantages of the prior art just mentioned by performing a dialogue by visual contact between a mirror surface and a user or a plurality of users. The invention achieves this goal by providing an interactive mirror surface method according to the appended claim 1.
    [0014] Computer vision technologies, and in particular view synthesis algorithms, make it possible to reconstruct in real time a new 3D visualization from a set of media sensors. This synthesis of the mirror reflection and the automatic monitoring of the eyes of the users are used in the invention to visualize a mirror reflection adapted so as to stimulate a visual contact between the mirror surface and a particular user. This new communication facility is used throughout the duration of a usage session. The mirror surface, thanks to its 3D audio-visual rendering and in particular thanks to its interactions, plunges the user into an immersive universe.
    According to an advantageous embodiment of the method of the invention, illustrated in FIG. 2, the mirror can capture the attention of a user by creating eye contact with it. Once it is acquired, the interface of the mirror surface is personalized for the user and he can then consult his information displayed on the screen while keeping a visual contact with his mirror reflection. In particular, he has the freedom to move in front of the mirror surface by interacting and while keeping a visual contact with his image reflected in transparency with the applications of the mirror surface.
    The space required to implement the visual communication system is reduced compared to the dual use of a physical mirror and a visual communications system.
    In addition, in accordance with another advantageous mode of implementation, illustrated in FIG. 3, the construction of the mirror reflection of a user can be customized with respect to one or more references that are in the immersive environment. In this case, the angle and the point of view used to build the mirror reflection is made with respect to this reference or references and with respect to the user himself. Typically, the user can, for example, be seen from all angles by varying the angle of view interactively from 0 ° to 360 ° by moving around a reference placed in the center of the media sensors.
    BRIEF DESCRIPTION OF THE FIGURES
    The invention will be explained in more detail with the aid of several examples of embodiments presented by the following drawings:
    Fig. 1 represents a flow diagram illustrating an example of the method of the invention.
    Fig. 2 illustrates the example of an embodiment of the initial phase of establishing a two-way visual contact between the system and a particular user.
    Fig. 3 illustrates an example of an embodiment with the use of a reference in the interaction space. DETAILED DESCRIPTION OF THE INVENTION
    The features described in claim 1 allow to establish a communication and intuitive man-machine interactions with a user but also in the case where several users could potentially dialogue through visual contact with a mirror surface.
    In the method of the invention, an "s" should be added to man in the terminology "human-machine interaction" to emphasize that a plurality of users can interact.
    The invention uses the mirror reflection of users to attract their attention and to invite them to interact with the mirror surface. Throughout the use session, the user keeps this visual contact by viewing his mirror reflection. A new communication link is thus created between the user and the mirror surface but also with the spectators nearby. Indeed, these can perfectly identify and visualize between which the communication is established, this thanks to the visual contact established between the user and its mirror reflection.
    The following definitions can better describe the underlying concepts of the invention.
    Definition: "Computer system" means a set of interconnected and powered electronic components which have among them, at least one processing unit (processor), inputs and outputs, memory for storing the information and the data. software to run the applications. Computer systems are preferably distributed, that is, they are interconnected by a computer network and their software, which controls them, communicate with each other. Computer systems, sometimes called connected objects, come in a multitude of forms and appearances, such as: a computer, a smartphone, a watch. As embedded computer systems, they can also be integrated in the clothes we wear or in the environment around us and this without necessarily being visible.
    Definition: the "geometric characteristics of the mirror reflection" means the angles of view (ΘΓ, φΓ) of azimuth and elevation and the coordinates of the point of view (xr, yr, zr) and the angle of field ar; in other words, the set of parameters that a virtual pan-tilt-zoom camera should have to capture the image of the mirror reflection from any point in a 3D space.
    Definition: "visual characteristics" of an application means the construction of a 3D view of the graphic objects of the application with a shooting characterized by the angles of view (ΘΓ, φΓ), d ' azinnute and elevation and by the coordinates of the point of view (xa, ya, za) with a field angle aa.
    Definition: Mirror "graphic nature of the reflection" means all types of graphic visualizations, transformations, augmented reality additions, and digital filtering that can be applied to its graphical representation.
    An example of adaptation of the graphic nature of the mirror reflection is that which simulates the window of a store, when a person is located upright and outside the store. The image of the reflection is then often against the light, the person can nevertheless see inside the store. By adapting the graphic nature of the mirror reflection, the mirror surface can thus perfectly give the illusion to a user to be in front of a virtual storefront.
    Another example of adaptation of the graphical nature of the reflection makes it possible to communicate emotions between the reflection of a person and himself, for example when the reflection blushes or by a more complex process of visual transformations, such as this one. described in the publication (Dynamic 3D Avatar Creation from Hand-held Video Input, by Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly, ACM Transactions on Graphics [Proceedings of SIGGRAPH], 2015).
    Definition: by "synthesis of the mirror reflection" is meant a processing chain that makes it possible to construct a new 3D visualization or a 2D view thereof with the desired geometric characteristics and this from the existing views which include, preferably the depth (see RGB-D images Red, Green, Blue and Depth). The depth defines the distance of the voxels from the image to the media sensor.
    A multitude of examples of synthesis algorithms of views exist, they represent indeed an important research axis in the field of computer vision and a large scientific literature exists on the subject. A taxonomy and an overview are described in the reference work: (Computer Vision: Algorithms and Applications by Richard Szeliski, Published by Springer, 2011).
    Definition: "visual contact" means the contact established when a user sees a graphical representation of his mirror reflection and when he has the perception that the reflection reacts according to his visual attention.
    In the context of the invention, the visual contact between a particular user and his reflection is used to allow several people to interact with the surface by identifying couples: user - reflection.
    Definition: "reported interaction" means one that uses a device connected by a computer network and communicates with the mirror surface.
    [0034] Definition: "visual contact descriptor" means the set of data which contain the user's identifier, the status of the type of visual contact in progress, and the history of the last visual contacts and the last interactions with the mirror surface.
    This descriptor can be customized with respect to particular users or adapted to particular applications; to take into account, for example, an interaction reported on another portable device or a connected watch. The treatment of visual contact can also be carried on the device if it has the necessary sensors.
    One or more users may be in visual contact with the mirror surface. They are then associated with the surface by one or more descriptors, unlike other users viewers who are not in visual contact.
    Definition: "use session" is the time during which a visual contact descriptor is assigned to a user, a user session has four phases: the inactive phase, the initialization phase, the phase during the session, and the phase of the termination.
    A usage session allows intra-session personalization of the mirror surface as opposed to an intersessional customization that requires the implicit or explicit identification of a user.
    Definition: "interaction space" means the space where a user can interact with the mirror surface; specifically, where he can make eye contact.
    Definition: "user reference" means the set of data extracted from the media stream and which contains information relating to the location, position, and orientation of a user's gaze, such as: identifier, the reference coordinates of the position of the user's head (xu, yu, zu), the angles of the orientation of the head (0t, φ ^, as well as the angles of his gaze (0U, φυ ) of azimuth and elevation.
    Definition: "media sensors" means the equipment that performs the acquisition of audio-visual signals necessary for the synthesis of the mirror or reflections and audio-visual rendering of the mirror surface. These signals are also exploited in the interpretation of human-machine interactions with the mirror surface.
    There is a wide variety of sensors that can achieve the invention with different qualities and performance. A classification of specific sensors for the construction of views is detailed in chapter 8 of the book (Computer Vision for Visual Effects, by Richard J. Radke, Published by Cambridge University Press) and the latest plenoptic camera technologies are described in Chapter 5, (Digital Representations of the Real World by Christian Theobalt, Olga Sorkine-Hornung, Oliver Grau, Marcus A. Magnor, Published by CRC Press). The evolution of plenoptic cameras makes it possible in particular to directly acquire the 4D information of a luminous flux (Light-field camera). These cameras offer the advantage of facilitating the computation of the stereo correspondence necessary for the synthesis of virtual views.
    Other panoptic-type sensors also make it possible to carry out the invention, for example, that described and filed in the patent (see Omnidirectional Sensor Array System, WO 2012 056 437 A1).
    Definition: "Media flow" means all the digital signals from the media sensors transported from them to the computing power that performs the vision processing and processing interactions.
    Definition: "self-visual content" means all of what a software application is likely to generate output and what the user can perceive, such as for example: text, images, media, music, a video game, and also sources of augmented reality.
    Definition: "audio-visual context" means the scene where the user is placed without including the voxels of the user or users.
    Definition: "reference in the immersive environment" means any zone or point in the interaction space, the mirror reflection, an augmented reality representation, or an image displayed on the mirror surface.
    This reference may also be a fictitious point or a point or an area identified by a marker.
    In the context of the invention, the immersive interactions are also determined with respect to these references.
    Definition: "immersive interactions" means interactions where the user to the perception of being physically in the audio-visual context or audio-visual content with which it interacts.
    Definition: "Mirror surface" means one or more display screens 2D or 3D auto-stereoscopic type. This mirror surface can also be configured as an immersive room where the entire visual field of the user is recreated.
    Definition: "visual contact dialogue" means a human-machine interface that uses visual contact to communicate with a user or a plurality of users.
    Definition: by "physical marker" means a mark visible by a user, as opposed to a fictitious point which is not visible to him.
    Numerous technologies and screen formats exist in the state of the art and can advantageously exploit the invention. Different visual renderings exist, for example the auto-stereoscopic visual rendering and the dynamic perspective rendering which is realized by the projection of a 3D representation in 2D dynamically according to the orientation of the user's gaze. Although rendering with this projected view does not allow the user to perceive a 3D relief, it still has an interesting advantage compared to an auto-stereoscopic screen display: the visual quality is more constant compared to the displacements of the user.
    The method of the interactive mirror surface will be better understood with the aid of the illustration of the flow diagram of the method of FIG. 1.
    The method of an interactive mirror surface is decomposed into several steps.
    [0057] When a user has visual contact with the mirror surface, this visual contact is characterized by a descriptor. When a user enters the presence of the interaction surface, he does not yet have a descriptor assigned by the surface. Step executions capture the user's attention by stimulating eye contact. In step (a), when the user comes into visual contact with the surface and enters the media stream (f), step (a) assigns a descriptor to it and initiates a session of use. The user can then interact with the surface at will. On the other hand, when the user no longer wishes to interact with the mirror surface, he implicitly loses the visual contact descriptor and the use session is terminated.
    The user can also terminate the session of use explicitly by a human-machine interaction to allow another user to interact by visual contact with the mirror surface.
    In some cases, several users can interact with the surface with, for each of them, a visual contact descriptor is then created with an associated use session.
    A set of heuristics are used to remove a visual contact descriptor from an idle user during a usage session.
    A first heuristic simply detects if the user leaves the interaction space, in which case he implicitly gives way to visual contact. The second heuristic determines the inactivity of the user, and finally, the third determines the inattention of the user from the visual contact he has with the surface. The last two heuristics use the data from the visual contact history and interactions to determine the user's inactivity. This inactivity or lack of attention is also communicated to the user by adapting the graphical nature of the mirror reflection in step (c). Typically, if the user is not attentive, the intensity of the mirror reflection gradually fades as a function of the time that elapses. After some time has elapsed, the system attracts the user's attention proactively. In some embodiments, the heuristics are advantageously replaced by automatic learning methods where the system learns by itself to better dialogue through visual contact with the users.
    To perform the first step (a) of the loop, media sensors preferably arranged around the interaction space are calibrated and modeled, these procedures are illustrated in the following work: (Multiple View Geometry in Computer Vision - Second Edition by Richard Hartley and Andrew Zisserman, publisher Cambridge University Press, 2004).
    In step (a), the digital signals of the media stream (f) are processed to extract the following information: the automatic tracking of user repositories, the management of the visual contact descriptor (s) and all the man-human interactions. machine.
    When an event is detected in step (a) it is transmitted to the other steps (b) and (c) which reflect the implications in their processing.
    The module for the automatic tracking of user repositories is performed by an algorithmic analysis well known in the prior art of computer vision. A virtual mirror system was published in 1998 (see A Virtual Mirror Interface using Real-time Robust Face Tracking, by T. Darrell, Gordon G., J. Woodfill, M. Harville In Proceedings of the Third International Conference on Face and Gesture Recognition, April 14-16, 1998, Nara, Japan). This publication describes how to use stereoscopic vision to calculate the depth, the extraction of the person's outline from the audio-visual context, the monitoring of the position of the head and, finally, the composition of the video images with the possibility of 'bring some graphic transformations to the mirror reflection.
    Since this publication, many improvements have been made to the media sensors and the processing chain of the view synthesis. These improvements are typically interdependent; the choice of a type of media sensor partly determines the necessary processing chain, p. ex. when the depth data are made available to the view synthesis algorithm by the use of an active sensor, that is to say, with a sensor that projects light into the scene to be viewed, it must then align the different images between them. To perform this task, specific algorithms exist, such as p. ex. the different versions of ICP: Iterative Closest Point algorithm (see A method for registration of 3-D schapes by PJ Besl and ND McKay In IEEE Transactions on Pattern Analysis and Machine Intelligence, 14 (2): 239-256, 1992) .
    In some cases, some of the specific algorithms are made available by the designers of the media sensors, this is particularly the case for the Kinect ™ media sensor of the Microsoft® company.
    Recent approaches to automatic tracking algorithms typically exploit machine learning methods with large amounts of data, such as the Kinect ™ media sensor described in the publication (see Decision Forests for Classication, Regression). , Density Estimation, Manifold Learning and Semi-Supervised Learning, by A. Criminisi, J. Shotton, and E. Konukoglu, Microsoft Research Technical Report TR-2011-114).
    Step (a) also aims to analyze and treat the interactions of users who are in visual contact with the surface. The human-machine interface is preferably multimodal, that is to say, the mirror surface has several interaction modalities that the user can use separately or by combining several of them. This richness of interaction is advantageously exploited to make the mirror surface as user-friendly as possible. These interactions typically involve the analysis of gestures, speech, tactile interfaces. By the sub-step (a2), these interactions can even be reported on other connected devices, such as a laptop, a watch, or joystick for example.
    Step (b) is composed of the processing chain (see pipeline in the English terminology) necessary for the synthesis of the mirror reflection from the media stream (f). Step (b) also applies the necessary processing to software applications to output a virtual reality format. This format is then used to make an audio-visual composition in step (c). Some software, such as video games, directly build representations in an appropriate format of virtual reality.
    The processing chain of the synthesis of virtual views realizes the construction of the mirror reflection of one or more users and adapts the geometric characteristics of the reflection to attract and notified the user's gaze. As illustrated by the example of FIG. 2, the reflection is visualized so as to look at the user when it is not yet in visual contact with the surface. Otherwise, the user has a visual contact descriptor and the mirror reflection summary is then customized for that user during this session of use.
    To achieve this synthesis of views, a multitude of approaches are described in scientific publications, such as that published in the following work: (Image Processing, Analysis, and Machine Vision - Fourth Edition, by Milan Sonka, Vaclav Hlavac, Roger Boyle, published by Cengace Learning 2015). The processing chain is entitled: 3D reconstruction from an unorganized set of 2D views, and Structure from Motion pipeline. This example describes the set of a necessary processing chain as well as the type of format generated from a 3D point cloud converted by triangulation and then bringing the respective colors to each triangle. The format of the representation is made compatible with virtual reality software.
    In step (b), the calculation of the synthesis of the mirror reflection, which uses the media stream (f) and the data of step (a), is automatically optimized to use the least number of shots. necessary to build a mirror reflection turned towards the user. These optimizations are possible, because unlike a generic application of synthesis of views, the system knows a priori shooting to compose together.
    In some cases of synthesis of views, occlusions appear, they are parts of the image that are hidden by viewers or by the user himself. During a session of use of the surface, shots are recorded beforehand and may possibly reconstruct the desired mirror reflection with a better visual rendering, even if, instantly, this operation would have been impossible to achieve by the occlusion or occlusions .
    In step (c), when a visual contact descriptor is assigned to the user, some additional improvements are made possible by the personalization and the interactions with the user. In particular, the user repository tracking module and view synthesis may consider an additional reference defined in the immersive environment to compute the point and viewing angles of the mirror reflection. These interactions are typically immersive: the user somehow immerses himself with his mirror reflection or with the visualization of data from a software application. An example of these immersive interactions is illustrated by the embodiment of FIG. 3.
    Sub-step (b1) makes it possible to exploit the data recorded previously in the current use session or possibly during those previously recorded. To perform inter-session backups, the user must be explicitly or implicitly identified.
    The result of the synthesis module of the mirror reflection typically generates a 3D representation that can be viewed and processed by virtual reality software or simply projected views in 2D.
    In step (c), the 3D audio-visual composition of the sources is carried out. This module essentially uses the following treatments: the homographic transformations of the images to better integrate the different visual sources and the subtraction of the background scene which makes it possible to substitute another audiovisual context for that of the interaction space.
    When the mirror surface has allocated a visual contact descriptor to the user and initialized a session of use, the method of the invention has the advantage of offering personalized treatments to the user. These use the data collected during the current session and possibly those of previous sessions.
    According to the embodiments, it is advantageous to identify the users to further customize the interactions and data of the software applications. This identification is carried out in two ways: either explicitly by asking the user to disclose his identity or implicitly "on the fly", that is to say, without the user identifying himself, it is the surface that recognizes the user. In both cases, the goal is twofold, first, to improve the quality of the mirror visualizations and, secondly, to improve the interactions with the software applications by taking advantage of the profile, the usage statistics, and the preferences of the applications. the user.
    Step (c) performs the composition of the audio-visual sources. The sources are those of the surface itself, those of the mirror reflections and those of the applications that are executed on the mirror surface by the user.
    It may sometimes be interesting to dynamically modify the visual perspective of the audio-visual contents with respect to a user repository to maximize the quality of its user experience: p. eg, to make it easier to read a document if the user is moving or to place the user in immersion in relation to the information with which they interact. Step (c) thus performs the necessary transformations. These can be partly described by the metaphor of a fish tank that is used in the scientific literature to define this type of visualization: (see Fish Tank Virtual Reality, by C. Ware, Arthur K., and KS Booth. In INTERCHI'03, 1993).
    Another composition commonly used is to replace the audio-visual context with another. For audio, the operation is performed by 3D spatial rendering that achieves the desired goal. To reposition mirror reflection in another context, particular algorithms are needed. These are known in the state of the art (see Probabilistic fusion of stereo with color and contrast for bi-layer segmentation, by Kolomogorov et al., In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006).
    [0084] More complex compositions are defined by the English term environment matting to account for the light interactions between the different visual sources (see Environment Matting and Compositing, by DE Zonker, DM Werner, B. Curless, and DH Salesin, In ACM SIGGRAPH 1999 Proceedings Conference).
    Step (c) also adapts the graphic nature of the mirror reflection. This function is advantageously used by the invention to communicate at all times with the user in order to maintain visual contact with him in relation to the interactions he executes. A multitude of possibilities make it possible to visualize the user's attention to his mirror reflection or in relation to an application that the user executes on the mirror surface. For example, when the user seeks contact with his reflection it will be viewed in a complete manner while if the user pays attention to the results of a software application, the reflection becomes more transparent with, possibly display that comes down to the outline only.
    In some cases, the 3D type of audio rendering is used to reinforce the immersion impression and thus to advantageously complete the visual rendering of the mirror surface. This 3D audio rendering is decisive, if the audio-visual content of a software application lends itself to it, as p. ex. for video games, 3D movies, music concerts, or if creating a particular user experience is desired. If several mirror surfaces are made available to a set of users and in the same space, a sound shower can be advantageously used to enhance the experience of each user individually. A sound shower makes it possible to locate the audio diffusion with respect to a reduced space when it is shared by several installations of mirror surfaces.
    Those skilled in the art will understand that the scope of the invention is not limited to a type of media sensor or internal structures of the particular data, or even to a particular choice of vision algorithm. computer.
    Profitable developments arise from the independent claim for application contexts where one can validly offer the following uses: a physical mirror and a visual communication system. Beyond this potential, the invention establishes a new paradigm of human-machine interaction that makes it possible to attract the attention of a user by creating a visual contact with his mirror reflection and this for the different phases of a session of use: the initialization, the session itself, and the termination.
    Today, visual communication screens are blooming everywhere, in this context many new applications are made possible by the invention. There are places where the user will be naturally present with the two uses: that of a physical mirror and that of a communication system. These uses can be found in a private company such as in the lobby, in a hotel or in a private home.
    In these application contexts, the advantages that flow from the invention are multiple: [0091] A common advantage of the various embodiments is to offer an intuitive and natural means of attracting the attention of a particular person among other people. The English term of eye-catching defines this property; to achieve this objective, the mirror surface visualizes the mirror reflection of the people present, unlike a physical mirror, the reflection follows the person's gaze and invites it to come into visual contact.
    This mechanism establishes a visual communication and captures the attention of a person and then establish a visual dialogue between it and the mirror surface. In particular, after the initialization phase by eye-catching, the session of use then exploits the visual contact and the associated interactions. When the visual contact with the user is broken, the session of use enters the phase of termination and, again, the communication by visual contact is put to advantage.
    Another advantage offered by immersive interactions is the improvement of the user experience; the system makes the most of the capabilities of human perception. Typically, the more the expressiveness and the dimensions of the surface are close to human perceptive capacities, the better the user experience; indeed, the surface can thus visualize and make the best use of information exchanges with the user. Beyond the dimension of the surface itself, the immersive interactions bring an additional dimension - immersion - which further increases the perceptive space made available by the surface.
    When the interactions are placed in a perceptual space close to human possibilities, the quality of visual communication is optimal. This advantage is illustrated here with the example of a search engine. Indeed, unlike the current user interfaces where the results are visualized by a list of texts or an array of images, the interactive mirror surface offers an immersive approach where the user "enters" in some way into the virtual space of data. The structures of the visualized data can be enriched compared to a simple textual list, the scheduling of the relevant results can also fully benefit from the expressiveness offered by the immersive interactions.
    Another advantage that follows from the invention is the space saving achieved by offering both uses, the physical mirror and a visual communication system on one and the same mirror surface.
    Another advantage of the mirror surface is to provide a human-machine interface where several users can interact while knowing exactly who interacts with the surface through the visualization of the mirror reflection of the interlocutor or with whom the surface communicates. Spectators thus participate in the new communication channel created by the visualization of visual contact.
    In other embodiments, a single mirror surface can be shared between several users. The surface can also be distributed visually on several separate screens.
    According to the embodiment, the computing power required to calculate the mirror reflection can be made available locally or remotely in the cloud, typically. The same architecture is conceivable for storing the data, these can be available locally or remotely, connected by a computer network.
    Making calculations and remote storage of data by pooling computer resources allows, for some embodiments, to significantly reduce costs and makes the invention all the more commercially interesting.
    EXAMPLES OF EMBODIMENT OF THE INVENTION
    The first embodiment is where the users are located in places where they spend a few moments either to wait or to inquire, this ephemeral mode of use is illustrated in FIG. 2. This mode illustrates the initialization of a dialog by visual contact. A second embodiment is illustrated in FIG. 3 where the interactions are described and this in the context of a tattooist shop.
    The first embodiment of FIG. 2 illustrates the initialization phase of a usage session. The application context is that of an entrance hall of a hotel. In these places, the two uses of a physical mirror and that of an information system are frequently found. Today, without having an interactive mirror surface, information on tourist and cultural activities is often communicated with paper leaflets.
    The interactive mirror surface (210) includes the surface itself (212), typically an auto-stereoscopic screen with an audio amplification system and loudspeakers, a computer with computing power local (214) and a database (213) which contains the recorded data of hotel guests. The mirror surface displays the reflection (211) of the user (201). The media sensors are installed (220) to acquire the media stream and to build the 3D image of the user's mirror reflection so that the person can make visual contact with his mirror reflection (211).
    In a first step, the user (201) walks in a given direction (202) without paying attention to the mirror surface (210). When the user enters the interaction space (200), the reflection of the user is visualized in step (c) of the method of the invention, this reflection is constructed to look towards the user (201). ) to get his attention and thus to create eye contact. The user (201) then looks at his reflection (211) by turning his head in the direction of the reflection. From this moment, the visual behavior of the surface changes and allows the user to interact in a personalized way. In step (a) of the method, a visual contact descriptor is assigned to the user and a usage session is initialized. With the approach of the 3D reconstruction of mirror reflection and unlike other types of mirror, the user does not need to be positioned in front of the mirror surface to see face to face or to interact with it.
    During a session of use with the mirror surface, the user could typically request by voice interactions an Italian restaurant proposal for the evening. The audiovisual content of the mirror surface is then enriched with different restaurant proposals, while displaying the mirror reflection (211) of the user in transparency and with a front view. The graphic nature of the reflection is now adapted to allow easy reading of the visual content relating to restaurants, the contour of the face of the user is visualized only. The transformation and filtering of the mirror reflection are carried out in step (c) of the method of FIG. 1. Finally, when the user has read the displayed information and is about to leave the interaction space, the mirror surface detects, in step (a), the termination of the session of use, the visual contact descriptor is deleted. The method of the invention continues while waiting for another user to come into visual contact with the mirror surface.
    If the user desires, he can identify himself and register on the surface via his laptop using an application on his laptop provided for this purpose. He can also benefit from a richer personal interaction by sharing his personal data from his laptop with the mirror surface.
    In this embodiment, several people could potentially attend the interaction with the mirror surface without necessarily taking part in it.
    A second example of embodiment is that of the stall of a tattoo artist, so the mirror surface and its interaction space are illustrated in FIG. 3.
    In particular, the immersive interactions of the user make it possible to control the point-of-view and the pan-tilt-zoom parameters of the user's reflection with the additional advantage of being able to frame the vision of the mirror reflection on a part of the image. accurate of the human body.
    The idea is to exploit the invention in an augmented reality application context to allow to try a tattoo before getting tattooed. A software application is also proposed on the mirror surface to facilitate the search and the discovery of a tattoo that corresponds to the wishes. The user can even, if necessary, recreate, modify, or customize a tattoo with the use of this software application.
    [0110] The state of the art mentions the possibility of trying on clothes before buying them or hairstyles. Without doubt, the most sensitive application is that concerning tattoos, it is difficult to go back once the tattoo is tattooed. It will therefore be particularly advantageous to use the invention to visualize a tattoo. The invention allows to visualize a tattoo on all parts of the body, such as for example, on the lower back without having to contort. With augmented reality visualization, all geometric parameters of the tattoo can be adjusted, eg. ex. size, orientation, position. The invention makes it possible to see certain parts of the body that are difficult and sometimes impossible to look at with another type of mirror.
    The interactive mirror surface (310) is composed with the following elements: the surface which is typically a human-sized auto-stereoscopic screen with integrated loudspeakers, a connection to a computer communication network (313) The purpose of this connection is to carry out the majority of calculations and data storage in a shared IT infrastructure. The interaction space (300) is surrounded by media sensors (314), which are necessary to have a realistic reconstruction of the mirror reflection from all angles of view. High performance video systems typically include a sphere with a hundred media sensors.
    The interactive mirror surface method is shown in the example of FIG. 1. The illustration of fig. 3, presents the different elements that make up the use of the mirror surface and the immersive environment. When the visual contact is already established with the user (301), that is to say, the initialization phase of the session of use is already carried out, the customer of the stall now has the control of the interactions immersive, unlike the tattoo artist (302) who is simply a spectator. Only the person (301) has visual contact with his reflection (311) and a visual contact descriptor associated with the mirror surface.
    The immersive interactions are determined with respect to a reference (320) located in the interaction space. The mirror reflection is constructed in step (b) of the method of FIG. 1. The particularity of this embodiment of FIG. 3, is that the geometric characteristics of the mirror reflection are determined by the user reference and with respect to a first reference (320) which is in the interaction space. The mirror reflection is constructed relative to this reference as if the user were virtually replicated at the precise location of that reference. By moving around this reference, the user can see himself from all possible angles while watching his reflection on the surface. The user can therefore control the geometric characteristics of the mirror reflection by moving around this first reference. It can also control the elevation angle by looking down or up on the mirror surface. This construction of the mirror reflection is illustrated in FIG. 3. It allows the user (301) to see himself in profile on the mirror surface, while looking from the front mirror reflection. This feature of the invention is particularly advantageous because a physical mirror does not allow to see itself correctly in profile.
    In the embodiment illustrated in FIG. 3, the user (301) therefore has the possibility of moving relative to the reference (320) and in the different directions of the arrows (303). These displacements are reflected in real time on the mirror surface by the construction of the mirror reflection.
    In this case, the user (301) can move around the reference point to control the angle of view of the mirror reflection and thus it can vary the angle of view between 0 ° and 360 °. It can also move according to the arrows that go from left to right (303), moving closer to the reference point, it enlarges the details of the image by zooming in, moving away from the reference point, it zooms out. The geometric characteristics of the mirror reflection are thus modified in real time by the movements of the user in the interaction space and by the orientation of the gaze that it bears on its reflection.
    The two concentric circles (321) and (322) define the limits in which the user can influence the functions zoom in and zoom out.
    In this embodiment, the visual contact between the user and the surface is therefore permanent. Immersive interactions are composed of displacements and the orientation of the user's gaze.
    In the context of the stall of a tattoo artist, tattoos are preferably placed on the body and more rarely on the face. The user can customize the eye contact with a second reference. This will be advantageously placed where the tattoo will be tattooed. In this way, the eye contact is now transferred from the face to the place where the tattoo will be placed, that is to say, on the second reference which is located on the user himself.
    In the embodiment of a tattooist's shop, this second reference is typically identified by a physical marker applied to the body of the user. The mirror reflection is then constructed and personalized with respect to the two references simultaneously; the first being placed in the interaction space (320) to control, by the movements of the user, the viewpoint and pan-tilt-zoom parameters of the mirror reflection and the second reference, that of the physical marker, allows to automatically postpone the refraction of the mirror reflection and, at the same time, the visual contact on the future tattoo.
    Subsequently, this physical marker allows the tattooist to pinpoint exactly the precise point where the tattoo will be positioned, first by augmented reality, then by the actual completion of the tattoo.
    With both references, the user can now watch the tattoo placed on the lower back from all angles by moving around the first reference (320), with the mirror reflection which is framed on the height of the marker physical.
    In the example of this embodiment, several problems arise. The first is that of the occlusions created by the presence of the tattoo artist (302) who is also present in the interaction space. The solution proposed by the invention is to record images previously and use them to recompose the missing visual parts due to occlusions.
    Depending on the situation and the recorded data, a complete computer model of a person can be recreated, the person can visualize his model on the mirror surface with the interactions captured from the interaction space. This model can also be visualized later by software capable of displaying files with a format specific to virtual reality, the model can then be published and viewed on a website.
    In another use case, the person may wish to view the tattoo on another person than herself. This interaction is then performed using the records of another person's media feed, and an infographic template is created for that other person. The user present in the interaction space can view in
    权利要求:
    Claims (11)
    [1]
    real-time added augmented reality tattoo on the other person's infographic model while interacting in the interaction space, as shown in fig. 3. [0125] In some cases, the person preferred to view himself in a different audio-visual context than that of the tattoo artist's stall. These other contexts can be either previously stored or created on the fly to be subsequently composed with the voxels of the person, this visual composition corresponds to step (c) of the method of the invention. The visual composition will typically be adapted according to the light conditions to obtain a quality rendering. In other cases, when the use of the mirror is used at the same time as the software application design tattoos. The composition of the visual sources and the mirror reflection of the user are visualized with different perspectives. This situation is used by default, but it is not always desired. To harmonize the geometric characteristics of the transformations are necessary, the first typically consists of placing the image resulting from a software application in the same 3D space as that of the mirror reflection with the coordinates that define the position and the orientation as well as the size and the perspective of the image inlaid. The aim is to harmonize the visual composition of the two uses: mirror and application. In some cases of use, the two views, that of the mirror reflection and that of the tattoo design application are made integral in the 3D space. The coordinates of the mirror reflection are then moved p. ex. on the left of the surface to create room on the right to view the tattoos offered by the design application. The 3D mirror reflection and the other 3D images are then grouped in the same 3D graphic object. The immersive interactions of the mirror surface then directly control the entire composition of these two visual sources. In the example of the embodiment of the stall of a tattoo artist, it is interesting to apply a 3D transformation on the tattoo so that it perfectly matches the surface of the skin. The 3D computer model of the person allows precisely this transformation with the help of the physical marker that identifies the precise location where the tattoo will be tattooed. The application then makes it possible to better compare the different tattoos with appropriate transformations that make the tattoos realistic. The illustrated embodiments only show a subset of the range of advantageous developments of the method of the invention. claims
    A method of an interactive mirror surface being implemented using a computer system comprising an audiovisual interface, media sensors and an interaction space, comprising at least one sequence with the following steps: a function of the media flow of said media sensors and for each user present in said interaction space, processing the human-machine interactions, updating the user repository and the usage session; b) based on the data of step (a) and said media stream, constructing an audio-visual representation of the applications executed on said mirror surface and constructing the visual representation of the mirror reflection necessary for the visual contact dialogue; c) based on said data of step (a) and said media stream, composing audio-visual sources constructed in step (b), and then rendering audio-visual on said mirror surface.
    [2]
    The interactive mirror surface method according to claim 1, wherein step (b) comprises a substep (b1) which uses stored data of said media stream to construct said mirror reflection.
    [3]
    The interactive mirror surface method according to one of the preceding claims, wherein step (b) comprises a substep (b1) which uses stored data of said other person's media stream to construct said mirror reflection.
    [4]
    The interactive mirror surface method according to any one of the preceding claims, wherein step (c) composes said mirror reflection with a different audio-visual context than said interaction space.
    [5]
    The interactive mirror surface method according to one of the preceding claims, wherein step (a) comprises a substep (a1) where a user is identified.
    [6]
    The interactive mirror surface method according to one of the preceding claims, wherein step (a) initiates a session of use during visual contact.
    [7]
    The interactive mirror surface method according to claim 6, wherein step (b) modifies the graphic nature of said mirror reflection during said use session.
    [8]
    The method of an interactive mirror surface according to claim 6 or 7, wherein step (b) constructs the geometric characteristics of said mirror reflection together with said user reference and with respect to at least one reference in the immersive environment.
    [9]
    The interactive mirror surface method according to claim 8, wherein said at least one reference in the immersive environment is a physical marker.
    [10]
    The interactive mirror surface method according to one of claims 6, 7, 8 or 9, wherein said user reference system determines the perspective used in the audio-visual composition of step (c).
    [11]
    The interactive mirror surface method according to one of the preceding claims, wherein step (a) comprises a substep (a2) which allows another apparatus to connect and interact with said surface. mirror by postponed interaction.
    类似技术:
    公开号 | 公开日 | 专利标题
    US10347028B2|2019-07-09|Method for sharing emotions through the creation of three-dimensional avatars and their interaction
    US11043031B2|2021-06-22|Content display property management
    US10127632B1|2018-11-13|Display and update of panoramic image montages
    WO2019183276A1|2019-09-26|Augmented reality and messaging
    US20140267599A1|2014-09-18|User interaction with a holographic poster via a secondary mobile device
    US10580040B2|2020-03-03|Methods and systems for real-time image and signal processing in augmented reality based communications
    FR2917931A1|2008-12-26|METHOD AND SYSTEM FOR CONNECTING PEOPLE IN A TELECOMMUNICATIONS SYSTEM.
    US11150724B2|2021-10-19|Avatar-based augmented reality engagement
    CN113228625A|2021-08-06|Video conference supporting composite video streams
    BE1022580B1|2016-06-09|Method of obtaining immersive videos with interactive parallax and method of viewing immersive videos with interactive parallax
    US20210150197A1|2021-05-20|Image generation using surface-based neural synthesis
    CN109074680A|2018-12-21|Realtime graphic and signal processing method and system in augmented reality based on communication
    CH711803A2|2017-05-31|Interactive mirror surface interaction method
    Du2018|Fusing multimedia data into dynamic virtual environments
    EP2987319A1|2016-02-24|Method for generating an output video stream from a wide-field video stream
    CA3022298A1|2017-11-02|Device and method for sharing an immersion in a virtual environment
    US20210248789A1|2021-08-12|Intelligent Real-time Multiple-User Augmented Reality Content Management and Data Analytics System
    WO2020128206A1|2020-06-25|Method for interaction of a user with a virtual reality environment
    Zhang2019|A QoE Model to Evaluate Semi-Transparent Augmented-Reality System
    Wen et al.2021|A Survey of Facial Capture for Virtual Reality
    WO2021154433A1|2021-08-05|Systems, methods, and media for displaying real-time visualization of physical environment in artificial reality
    同族专利:
    公开号 | 公开日
    CH711803B1|2020-08-14|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    法律状态:
    优先权:
    申请号 | 申请日 | 专利标题
    CH01702/15A|CH711803B1|2015-11-23|2015-11-23|Process of immersive interactions by virtual mirror.|CH01702/15A| CH711803B1|2015-11-23|2015-11-23|Process of immersive interactions by virtual mirror.|
    [返回顶部]